Field
Embodiments of the invention are generally directed to online communication. More specifically, embodiments of the invention are directed to a variety of techniques for managing the content of chat messages exchanged between users in online environments.
Description of the Related Art
Multiplayer video games and virtual worlds have rapidly become popular forms of online entertainment. Both generally provide users with access to a virtual environment where they may interact with one another. For example, avatars are frequently used to provide a graphical representation of each user present within a virtual environment and users interact with one another through their avatars. Users control the actions of their avatar to navigate the virtual environment, e.g., achieve gaming objectives as well as to communicate or chat with other users. Such an environment may be persistent where events occur and users interact with each other regardless of the presence of any particular user. Alternatively, such an environment may be game or session based, e.g., where a group of users participate in a match of a first-person shooter game.
Chatting is a prominent feature within these kinds of online environments. For example, a user may chat directly with other users present in the same virtual location by typing characters into a text-field on a user interface. To promote civility and safety among users, a service provider hosting an online environment may desire to prevent the use of obscene language or other inappropriate communication, particularly in online environments developed for minors. In such a case, chat software may filter chat messages by removing swear words, slurs, or other known terms of disparagement. Blacklist filtering does not detect semantically unsafe messages; it can only filter individual words. Because phrases going through a blacklist filter are allowed by default unless blocked by a specific word in the blacklist, this approach allows guests to say a nearly infinite number of phrases which, in turn, makes it vulnerable to guests saying unsafe words using alternate spellings (replacing “S” with “$” or “cks” with “x”).
Whitelist filtering allows only safe words (or phrases), which prevents unsafe alternate spellings, but like blacklist filtering, does not detect semantically unsafe messages which can be formed using safe words. Further, whitelist filtering blocks at the word (or phrase) level, so words which are safe in some contexts but unsafe in others are blocked.
While such whitelist chat is safe, it is limited in the number of phrases it allows. Severely impacting user expressivity.
To protect a user's safety in online worlds, providers engage in continuous monitoring and filtering of chat. At a base level, providers filter out and prevent the sending of obvious profanity. The challenge is that people are very good at getting around the sort of filters which are typically used to detect and block unsafe messages. Every time a guests finds a way around one of the filter entries, a new entry must be made to block the unsafe phrases they are now saying. The system ends up flooded with filter entries put into place after-the-fact (meaning the damage is done by the time the filters are updated), often with entries which exist only because it was a work-around to another filter and which no one has tried to get around since it was first introduced. The race to detect and block unsafe message is a continuous, expensive process and in the end never provides a reliably safe environment for our guests. Additionally, the more complex a system becomes, the longer it takes for a phrase to be validated.